Variational methods that rely on a recognition network to approximate the posterior of directed graphical models offer better inference and learning than previous methods. Recent advances that exploit the capacity and flexibility in this approach have expanded what kinds of models can be trained. However, as a proposal for the posterior, the capacity of the recognition network is limited, which can constrain the representational power of the generative model and increase the variance of Monte Carlo estimates. To address these issues, we introduce an iterative refinement procedure for improving the approximate posterior of the recognition network and show that training with the refined posterior is competitive with state-of-the-art methods. The advantages of refinement are further evident in an increased effective sample size, which implies a lower variance of gradient estimates.

Neural Information Processing Systems http://nips.cc/

This approach has made variational inference applicable to alarge class ofcomplexgenerativemodels. However,manychallenges remain.

We propose a scalable algorithm for model selection in sigmoid belief networks (SBNs), based on the factorized asymptotic Bayesian (FAB) framework. We derive the corresponding generalized factorized information criterion (gFIC) for the SBN, which is proven to be statistically consistent with the marginal log-likelihood. To capture the dependencies within hidden variables in SBNs, a recognition network is employed to model the variational distribution. The resulting algorithm, which we call FABIA, can simultaneously execute both model selection and inference by maximizing the lower bound of gFIC. On both synthetic and real data, our experiments suggest that FABIA, when compared to state-of-the-art algorithms for learning SBNs, $(i)$ produces a more concise model, thus enabling faster testing; $(ii)$ improves predictive performance; $(iii)$ accelerates convergence; and $(iv)$ prevents overfitting.

The factorized asymptotic Bayesian ( FAB) approach has recently been shown as a scalable model-selection framework for latent variable models.

We present a probabilistic framework for nonlinearities, based on doubly truncated Gaussian distributions.

Recently, considerable research effort has been devoted to developing deep architectures for topic models to learn topic structures.

Moreover, we develop stochastic gradient MCMC inference that is scalable to very long multivariate count time series.

Recently, considerable research effort has been devoted to developing deep architectures for topic models to learn topic structures.